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Abstract 



This study demonstrates that even a criterion-referenced test, In 
which Items are chosen from a single content domain, requires a close 
examination of construct validity. The Individual Consistency Index 
(ICI) is effective in detecting anomalous response patterns resulting 
fromsome misconception(s). The subset obtained by deleting the 
responses which were spotted by ICI showed a higher ^dimensionality . 
The same result was replicated by another dataset whose test was 
parallel but not identical to the previous one. Although computerized 
error diagnostic programs can identify misconceptions possessed by a 
student in the very specific domain of arithmetic, ICI can be applicable 
to more general domains and detect possible candidates tc route to the 
expensive error-diagnostic programs. 
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Replace reference of Tatsuoka, K. K. , & Tatsuoka M. M. (Research 
Report 81-4) with 

Tatsuoka, K. K. , & Tatsuoka, M. M. Detection of aberrant response 
patterns and their effect on dimensionality (Research Report 
80-4). Urbana, 111.: University of Illinois, Computer-based 
Research Laboratory, April 1980. 
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Spotting Erroneous Rules of Operation 



> by the Individual Consistency Index 

Introduction 

4 

Item^Response Curve Theory (IRT) has proved its important role in 
modern testing practices such as computerized adaptive testing, in which 
each examinee takes a different set of items. The student's ability 
level is estimated and located as a point on a continuum. However, a 
drawback of IRT models that are presently available in practice is the 
constraint of unidimensionality on datasets (Lord & Novick, 1968). 
Reckase (1979) warned of and demonstrated the serious consequences for 
parameter estimation resulting from the violation of unidimensionality. 
It has been observed that the scores obtained from most achievement 
tests, unlike ability tests, are affected by two or more latent trait 
variables. 

Moreover, Tatsuoka and Birenbaum (1979, 1981) showed that the 
dimensionaltiy of a dataset obtained from the middle learning stages, 
when the students are still far from mastery, was multidimensional even 
though the test items are clearly drawn from a single domain. Their 
result indicates that a close examination of the construct validity of 
criterion-referenced tests is necessary. 

Brown & Burton (1978) developed an error-diagnostic model for 
whole-number subtraction problems. Their model "BUGGY" showed that 
wrong rules can yield the correct answers in some test items. Birenbaum 
& Tatsuoka (1980) found that 1 - 0 scoring based simply on right or 
wrong answers caused serious problems when erroneous rule* of signed- 
number operations were used by many examinees. 

Using data from a 64-item test consisting of four parallel subtests 
of 16 items each, Birenbaum & Tatsuoka first did a principal components 
analysis on the original data — with the items scored 1 or 0 in the 
usual manner. Next, the data were modified by giving a score of 0 when 
an item was correctly answered presumably by use of an erroneous rule, 
and another principal component analysis was done. The change between 
the two analyses was dramatic. The dimensionality of the data became 
much more clearcut with the modified data. The item-total correlations 
became much higher, while the means of the 16 tasks (each represented by 
four parallel items) did not change significantly. The above phenomenon 
anggests why some achievement tests cannot by treated as unidimensional 
even though the items are taken from a single content domain. 

One of the purposes of this paper is to investigate methods of 
extracting a unidimensional subset from an achievement dataset. Error 
analysis, which is usually performed by a series of clinical interviews 
and intuitive interpretation of the student's responses to the test 
items is a cumbersome work. At present, there are only a very few 
computer programs available for providing diagnoses of misconceptions 
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possessed by students, such as Brown and Burton's "BUGGY", and "SIGNBUG" 
developed by Tatsuoka <ai, , (1980). But they are expensive, and 
moreover, they can handle caly specific areas of arithmetic. Our 
intention is to find a more general technique applicable to other 
content areas in order to detect aberrant response patterns which are 
derived from erroneous rules of operation (or bugs). 

The second purpose is to replicate the result described in the 
Birenb&um-Tatsuoka study by applying the general technique for detecting 
aberrant response patterns derived from wrong rules of operation. 

It turned out that the index, Individual Consistency Index 
(ICI) introduced in Tatsuoka & Tatsuoka (1980) is very effective for 
spotting erroneous rules of operation in signed-nurober computation 
problems: The responses yielded by wrong rules are charac; . rized by 
having low scores and high ICI values. Moreover, the subset obtained 
from the original dataset by deleting the subjects who have low total 
scores and high ICI values demonstrated exactly the same phenomenon, 
that is, the dimensionality of the subset became nearly unidimensional , 
as the modified data did. The structure of the subset in terms of 
cognitive performance is interesting. It consists of the responses 
produced by using the right rule and errors probably committed by 
students randomly or inconsistently. The result will be useful for 
understanding the meaning of dimensionality of achievement data. It 
also shows the importance of construct validity, even in criterion 
referenced testing of the cognitive aspect of performance, and that the 
traditional means of item analysis that are based on taking the 
variances of binary scores and content analysis into consideration are 
not enough for constructing test items that are capable of diagnosing 
misconceptions • 

Method and Procedure 
Is 1-0 scoring justifiable? 

A test containing 64 signed-number addition and subtraction 
problems, consisting of four parallel subtests of 16 item* e*ch, was 
administered to 127 eighth graders at a local junior high school after 
the instruction was completed. (This test will be referred to as the 
"November data" hereafter.) Each item in th* i^t was carefully related 
so as to maximize the capability of diagnosing erroneous rules of 
operation. In signed-number computation, 98% of students' responses are 
summarized by four types: adding or subtracting two absolute values and 
putting the sl g iv o f plu9 o r minus on answers. Nine problem types in 
subtraction and six in addition (see Appendix I) are the necessary 
minimum number of items in order to maintain the error diagnostic 
capability for providing a specific description of a vast majority of 
popular errors. 

Tatsuoka et al. (1980) developed an error-diagnostic system called 
"SIGNBUG" for signed number problems on the PLATO system at the 
University of Illinois. With this computer program, the performance of 
the 127 students was thoroughly analyzed. 
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The same test was administered to 180 seventh graders who were 
still far frca the mastery stage and exhibited a variety of confusion in 
the material, (Data from this test will be referred to as the "January 
dat?" hereafter). The responses to the items in the January data were 
also analyzed by "SIGNBUG" and their complete erroneous rules of 
operation with those from the November data are described in Appendices 
I and II. Actually we have fouud many mere erroneous rules, incomplete 
ones and those applicable only to addition problems, but the number of 
bugs in the list is limited, to the erroneous rules that appeared in 
subtraction problems. The addition items form another dimension by both 
a principal components analysis, and a multidimensional scaling 
(Birenbaum & Tatsuoka, 1980) after the modification procedure described 
earlier was taken. Therefore we chose the subtraction problems for 
our study. 



Insert Table 1 about here 

Table 1 is the list of the binary scores >on 15 tasks (i.e., problem 
types) yielded by uhe 45 bugs given in Appendix II. The rightmost 
column shows the total scores on 15 tasks. The first and second numbers 
in the parentheses are the total scores on addition and subtraction 
tasks, respectively. The bottom line of Table 1 contains the total 
number of l's for each item type. For example, for the task -16 - (-7), 
the'eorrect answer is yielded by 26 out of 42 erroneous rules. If the 
data is collected while many students are confused by a variety of 
errors, then this task, No. 8, will have the highest number of l's. 
Thus, although task No. 8 turned out to be the "easiest" in the January 
data, this must be partly attributed to the fact that the correct answer 
can be obtained by so many erroneous rules. Table 2 shows the fttuk 
orders in the proportion correct of 12 tasks for the January and the 
modified November data (as described earlier). However, the 



Insert Table 2 about here 
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counterpart of this task, -6 - (-8) has only twelve l's out of 42. 
These items should almost be equally diff'cult *Lth respect to the 
conceptualization of the subtraction problems because of the teaching 
method. But their positions of the item difficulty order in the 
datasets (both November and January data) are quite different. The 
descending order of the total l's over 15 tasks in Table 1 is 8, 15, 7, 
16, 11, 5. These six items are in the top seven items having the largest 
number of the total l's in the January data. As mentioned earlier, 80% 
of the examinees in the January data used a variety of erroneous rules 
of operation. 

Importance of Item Ordering 

Harnisch and Linn, (1981) classified indices that measure the degree 
to which an individual response pattern is atypical into two different 
types. One type consists o, those foruuiated by using the orders of 
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Table 1 



Observed Complete Rules and their Response Patterns 
for Signed Number Addition and Subtraction Problems 



Bugs 



Addition 



Subtraction 



Total 
Scores 
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5 


10 


11 


14 


15 


1 


2 


4 


7 


8 


9 


12 


13 


16 








1 or 2 








1 






1 


1 


1 


1 


1 


1 


1 


1 


1 


15 


(6, 


9) 


3 








1 






o 


1 


o 


1 


o 


1 


o 


1 


1 


11 


(6, 


5) 


4 


1 


1 


1 


1 






1 


0 


1 


0 


1 


o 


1 


1 


o 


11 


(6, 


5) 


5 


l 


l 


1 


1 


l 




0 


0 


0 


0 


0 


0 


0 


0 


0 


6 


(6, 


0) 


6 


1 




! 


1 






o 


o 


o 


1 


0 


o 


o 


o 


1 


8 


(6, 


2) 


7 or 8 


! 




! 


1 


! 
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1 


o 


1 


o 


1 


o 


o 


1 


10 


(6 


A ) 


9 


1 


1 


1 


1 


1 


1 


o 


o 


o 


o 


o 


o 


o 


o 


o 


6 


(6, 


0) 


10 


1 


1 


1 


1 


1 


1 


o 


o 


o 


o 


o 


o 


o 


o 


o 


6 


(6, 


0) 


11 


^ 


^ 


! 


1 


^ 


^ 


1 


o 


1 


0 


1 


0 


1 


1 


0 


11 


(6, 


5) 


12 


1 


1 




1 


1 


1 


1 


0 


1 


1 


1 


0 


1 


1 


1 


13 


(6 


»7) 


13 


1 


1 


1 


1 


1 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


6 


(6, 


0) 


14 






! 


1 




! 


1 


0 


1 


0 


1 


0 


1 


0 


0 


10 


(6, 


.4) 


15 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


0 


1 


14 


(6, 


3) 


16 








1 


! 


! 


0 


0 


0 


1 


1 


0 


0 


0 


0 


8 


(6, 


2) 


17 


1 




1 


1 


1 




1 


0 


0 


1 


1 


0 


0 


0 


0 


9 


(6 


>3) 


18 






! 


1 






0 


0 


o 


1 


1 


0 


0 


0 


0 


8 


(6j 


2) 


19 






1 


0 


1 




0 


1 


1 


0 


0 


1 


1 


1 


0 


7 


(2, 


5) 


20 








1 


! 
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o 
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o 


o 


o 


1 


8 


(4, 


>4) 


21 
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1 
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o 


9 


(6, 


i3) 


22 






1 


1 
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1 


o 
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1 


8 


(6, 


>2) 


23 


o 
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1 


o 




o 
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1 
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o 
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o 


5 


(3, 


2) 


24 


0 
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0 




0 
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0 
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0 


0 


0 


1 


5 


(3, 


,2) 


25 
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0 




o 
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o 
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o 
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5 
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27 
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o 
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2) 
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o 
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29 
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o 
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o 


o 
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1 
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o 
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o 
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3 
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|2) 


31 


0 
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o 
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(1, 


2) 


32 


0 
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o 
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i2) 


33 


0 
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o 
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34 
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o 
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o 


4 
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► 3) 


35 


0 
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36 
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0 
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1 


1 


1 


9 


(0 
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0 
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(3, 
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0 


0 
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(6 
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45 


1 


1 


1 


1 


1 


1 


0 


1 


0 


0 


0 


1 


0 


0 


0 


8 


(6 3 


,2) 



Total 1*3 24 28 26 30 26 32 12 10 10 20 26 10 10 9 18 

Proportion _3_7j>_9_J5l_l JL21^10.2026l£_10_9_13 

of 1 a to ^ 22 22 22 22 22 22 42 42 42 42 42 42 42 42 42 

# rtf Ktioa 



# of bugs 

rnir *22 erroneous rules (or bugs) because 21 out of 43 bugs used the right 

SbJM rule for addition problems. 



\ 

\ 



Table 2 

Rank Orders of the Task Difficulties (from Easy to Hard) 



a 

m Two Datasets 



Task !;°^^ e f January Data d 

Type Modified 



19 4- _ *J 




7 


— 1 *+ i — J 


Q 

j 


12 


— J t 1Z 


D 


D 


-5-H-7 


13 


3 


3 -h-5 


10 


11 


-6 + 4 


14 


5 


3-6 


6 


9 


-16 - (-7) 


7 


8 


-6 - (-8) 


1 


1 


-3 - +12 


12 


2 


2-11 


15 


10 


9 - (-7) b 


11 


4 


1 - (-10) 


4 




-7-9 


2 




b 

-12 - 3 


3 





a The task type 6+4 (item number 6 in Appendix I) has 
been omitted. 

These tasks were note included in the January test, 

°The November order is based on the difficulties estinated 
from the IRT model. 

^Tne January order is based on the actual proportion 
answering each item correct. 
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difficulty and the other comprises those based on the comparison of an 
individual response pattern to some kind of a standard response pattern. 
The former group consists of Van der Flier's index (1977) , the Norm 
Conformity index (Tatsuoka & Tatsuoka, 1980) and y index (Sato, 1972). 
The latter contains Sato's Caution index (Sato, 1975), Linn & Harniseh's 
(1981) modified caution index and Kane & Brennan's (1978) coefficient 
of agreement. A weighted sum of NCI leads to Cliff's (Cliff, 1978) 
Consistency Index, C t 2 (Tatsuoka & Tatsuoka, 1980) while the caution 
index has an algebraic relationship to Loevinger's homogeneity index 
(Takeya, 1978). 

Although this classif icatior is useful for pointing out a certain 
conceptual difference between the two types of indices, the fact remains 
that both types are dependent on the order in which the items are 
arranged in calculating them. This dependence on item order is made 
explicit in the case of the Norm Conformity Index (NCI), whose 
definition calls for arranging the items in descending order of 
difficulty for au arbitrary norm or reference group. The extent to 
which a given individual's response pattern then resembles a Guttman 
vector (in which all zeros precede all l's) with the same number of I's 
is what the NCI measures. The reference group may be the group of which 
the individual is a member, or It may be some other group of interest to 
l the researcher. 

On the other hand, the caution index was defined by Sato (1975) in 
the context of a data matrix in which the items are arranged In 
ascending order of difficulty for the group at hand and the individuals 
are arranged in descending order of total score. (Such a matrix is 
called an S-P table.) The question of dependence on item order does not 
normally arise since the caution index C^ is defined for this one 
* particular arrangement of items only. It is the complement of the ratio 
of two covariances that will be specified later. C± measures the extent 
to which the i-th individual's response pattern is atypiral of the group 
of which he/she is a member. However, there is no reason why we cannot 
speak of the atjrpicality of an individual's response pattern compared to 
the average response pattern of oome group other than the one to which 
he/she belongs. The order of items will then be different from before, 
and the value of C^ will change. Thus the caution index, too, is 
dependent on item order. 

We wish to demonstrate the extent and way in which NCI and are 
item-order dependent with reference to the set of 12 items shown in 
Table 2, that are common to both the November data and the January data, 
but have different difficulty orders in the two datasets. However, it 
may be useful first to give brief descriptions of the calculations for 
the two indices by using a smaller numerical example. ^ 

Example . Let us refer back to the 43 x 9 matrix of binary scores 
obtained by using the correct rule and 42 erroneous rules for solving 
nine subtraction problems, displayed in the right-hand panel of Table 1. 
We now pretend that this is a data matrix for a group of 43 examinees 
who took a nine-item test, and denote it as 
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Y - (yij), '1,2.. .,43; j-l,2,...,9 

Each element of the vector of column totals [y.i, y^*"' ?.9l is one 
greater than the corresponding column total shown in Table 1 (because 
the totals there excluded the I's in the first row since that row 
represented the correct rule of operation, whereas Table 1 was concerned 
with 1'8 generated by erroneous rules). However, since we need to 
arrange the items in monotonic order of difficulty (descending order for 
NCI, ascending for C^), let us renumber the items from 1 to 9 in the 
order they are to occijr in the formulas. 

For calculating NCI, then, the item order is (2, 9 , 13 , 12 , 4 , 1 , 
16, 7, 8), We take the second row (for Bug #3) as the response pattern 
of the examinee whose NCI we want to calculate. When the items* are 
rearranged as just indicated, the response vector becomes 

2 3 - [1 1 1 0 0 0 1 1 0] , 

whose elements we denote as y$\ 9 y32>»»»»y39 in ttie order they occur 
here. 

We also need the total score y3., which is 5 in this case. 
In general, 

j*l 

where n (- 9 here) is the number of items. The NCI for individual i may 
now be defined as 

(NCI)! - 2U la /U 1 - 1 , 



u " •% j J+ . a - i J >yik 



Ui - yi.(n-yi.) 

[Verbally, Ui a means the sum of all I's to the right of each 0 in the 
vector yi, added over all 0's; this represents the number of (0,1) 
pairs that occur in y^. Ui, on the other hand, represents the total 
number of (0,1) and (1,0) pairs in y^.] 
For our example, 

U3 a - 2 + 2 + 2- 6 

(because there are three 0's, each with two I's to its right; the last 0 
does not have any number to its right and hence contributes nothing to 
tut sua), while 



where 



and 
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U 3 - <5)<4) - 20 . 

Hence , 

(NCI) 3 - (2)(6)/20 - 1 - -.4 

(The negative sign indicates that the response pattern is closer to 
the reverse Guttman vector [1 1 1 1 10 0 0 0] than to the Guttman 
vector [0000 1 1 1 1 1], which agrees with our impression on looking 
at y 3 .) 

We now illustrate the calculation of for the sane response 
pattern. Since the formula calls for the items to be arranged in 
ascending order of difficulty, we reverse the response vector to get 

2 3 - [0 1 1 0 0 0 1 1 1] 

We also need the vector of column totals 

in - Iy.it y.2.«««» y.9l 

- [16, 20, 18, 12, 11, 10, 9, 9, 2] 

Finally, the reverse Guttman vector i-ith five l's is 

23 s - [1 1 1 1 1 0 0 0 0] 

Wit 1 =» elements of these three vectors appropriately symbolized, 
and '3' -t ed by 1 for generality, the formula for the caution index 
is 

Cl - 1 - cov <yn» y.j) 

cov (yfj, y.j) 

Making the substitutions, ve get 

9 9 

cov(y 3J , y.j) - [t^jy.i - ya. j^y.j 79 ] 78 

- [(0)(26) + (1)(20) + (1)(18) + ... + (1)(2) 
-(5)(26 + 20 + 18 + ... + 2)/9] # 

- -.875 

and 

9 9 

cov(y 3 j, y.j) - [^ySjy.j - y§. J 2 i y.j /9 3 /8 

14 
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- I(D(26) + (1)(20) + (1)(18) + (1)(12) + (1)(11) + 0 

-(5)(26 +20+18 +...+ 2)/9]8 

- 2.75 

Hence, C 3 - 1 - (-.875)/2.75 - 1.318 

Item-Difficulty Orders in Early and Late Le arning Stages. 

We now examine the response patterns of three students taken from 
the January dataset, exhibit their NCI and Ci values with items arranged 
in difficulty order for that dataset as well as in difficulty order for 
the November dataset , and note the extent and nature of the differences 
between the corresponding values. Although the two datasets are based 
on different samples, the results nevertheless give a general idea of 
the effect of using the difficulty orders in early and late learning 
stages. The relevant information is summarized in Tables 3 and 4. 



Insert Table 3 about here 

Student #37 is a very good student , who did most of the addition 
problems correctly, as seen in Table 4. About the only trouble she had 
was confusing parentheses with absolute-value bars, which is a 
relatively minor and easily remedied misconception. Yet, precisely 
because her misconception is a sophisticated one for students at this 
stage of learning , her response pattern is rare and atypical of the 
group. She thus gets a low NCI (-.10) and a high C x (.93), which are 
misleading because they imply that she needs to be cautioned and given 
remedial work. 



Insert Table 4 about here 

On the other hand, Students #12 and #30 possess misconceptions that 
are rather common in this dataset. Hence, their NCI values are 
relatively high (.62 and .69 , respectively) while their caution indices 
are moderate and low(.43 and . 16) v so that these students are net 
flagged for further attention. Yet, these students (like many others in 
the January dataset) had considerable trouble with addition problems, as 
evident from Table 4. Therefore, their not being cautioned was 
inappropriate, and was due only to their errors' being fairly common for 
their group. 

To confirm the above interpretations, let us now look at the KCI 
and Ci values for these three students when the items are arranged in 
difficulty order for the November dataset as modified in the manner 
described earlier. That is, we now inquire how anomalous each of these 
Students' response patterns would look if they had been members of the 
November group, who were close to mastery stage. The answer is, the NCI 
for Student #30 becomes the largest of three (.67), while Students #37 
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Table 3 

Response Patterns of Four Subtests for Students #37, #12 and #30 



January Order 


S#37 


S#12 


S#30 


Hov. Modified 


S?37 


S#12 


S#30 


i 

-16 - (-7) 


8 


0000 


1110 


1111 


12 + -3 


3 


1111 


1010 


1111 


2-11 


16 


0100 


1110 


0000 


-14 + -5 


10 


1111 


0010 


0000 


12 + -3 


3 


1111 


1010 


1111 


-3 + 12 


5 


1011 


1111 


1111 


8-6 


7 


0100 


1111 


1111 


3 + -5 


11 


1111 


1111 


0111 


-6 + 4 


15 


1111 


1101 


1111 


-6 + 4 


15 


1111 


1101 


1111 


-3 + 12 


5 


1011 


1111 


1111 


8-6 


7 


0100 


1111 


1111 


3 + -5 


li 


1111 


1111 


0111 


-16 - (-7) 


8 


0000 


1110 


1111 




in 


1111 


0010 


0000 


-O - \ O) 


1 


0000 


1010 


1000 


-6 - (-8) 


1 


0000 


1010 


1000 


-3 - +12 


13 


0111 


0100 


0000 


-7-9 


2 


1111 


0000 


0000 


2-11 


16 


0100 


1110 


0000 


-3 - +12 


13 


0111 


0100 


0000 


1 - (-10) 


4 


0000 


0000 


0000 


1 - (-10) 


4 


0000 


0000 


0000 


-7-9 


2 


1111 


oobo 


0000 


Total Score 




28 


27 


24 






28 


27 


24 


Caution Index 




.93 


.43 


.16 






.30 


.60 


.44 


NCI 




-.10 


.62 


.69 






.49 


.44 


.67 


ICI 




.96 


.88 


.99 






.96 


.88 


.99 



Table 4 

Error Patterns for Subject #37, #12 and #30 



Subject 


Addition 


Subtraction 




Subt 1 


Subt 2 


Subt 3 


Subt 4 


Subt 1 Subt 2 Subt 3 Subt 4 


S# 37 


(11,21) 


(11,0) 


(11,21) 


(11,21) 


treat parentheses as absolute value 


S# 12 


(13,21) 


(13,0) 


0 


(0,21) 


(13,0) (0,24) (13,0) 0 


S# 30 


(13,0) 


(13,21) 


(13,21) 


(13,21) 


(13,0) *(13,21) *(13,21) *(13,21) 



q * - (32) 
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and 12 have similar values (.49 ^nd .44). Their caution indices are 
.30, ,60 and .44 for Students #37, 12 and 30, respectively. As expected 
#37 has the smallest caution index, reflecting the fact, already 
mentioned, that her errors were more typical for students close to the 
mastery stage than the beginners in the January dataset. 

The foregoing illustrative examples should have underscored the 
undesirability (for error-diagnostic purposes) of the dependence of such 
indices as NCI and Ci on the particular difficulty order of items that 
is used in computing them. Clearly, what is needed is an index that 
does not depend on the item difficulty order for a group but on that for 
each individual in his/her own right. 

The Individual Consistency Index (ICI) 

The Individual Consistency Index (ICI) depends on the task 
difficulties as determined by an individual student's state of 
knowledge. Its definition calls for the existence of two or more 
parallel subtests, and the sets of parallel items are arranged in 
ascending order of task diificulties for the student in question — 
i.e., in the order of the student's subscores on the separate tasks such 
as the 12 shown in Table 3, where each task is represented by four 
parallel items. The actual calculation of ICI is the same as that for 
NCI — or, more precisely, for a group of m subjects, where m is the 
number of parallel subtests. 

Its value is large to the extent that the individual remains in the 
same state, and hence responds to similar items in the same way — i.e., 
using the same rule. If ICI is low, on the other hand, then the 
individual is probably not sure how the problems can be solved so he/she 
tries various strategies to respond to the questions. Or, he/she is 
careless and makes a considerable number of random errors. Thus, ICI 
can serve better than NCI and the Caution Index as an index for flagging 
individuals who probably require further attention for remediating their 
errors and/or making finer diagnoses. 

Table 5 is a summary of the error patterns committed by the 127 
Students in the November data, divided into four subgroups by the 
following criteria: (1) students whose scores are higher than 52 (the 
highest total score in Table 1 13. Therefore 4 x 13 is taken as a 
criterion) and ICI values are greater than or equal to .90; (2) those 
whose scores are lower than or equal to 52 and ICI values are greater 
than or equal to .90; (3) those who have scores higher than or equal to 
52 and have ICI 



scores 



3 


1 


4 


2 



.90 0 



52 
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values lower than .90; (4) those whose scores are lower than or equal to 
52 and ICI value? are lower than .90. 



Insert Table 5 about here 

The error types and their frequencies are shown in four sections of 
the table. As stated in the table, a number sign (#) in front of an 
error pattern represents the scores adjusted for both addition and 
subtraction problems (Blrenbaum & Tatsuoka, 1980). The question marks 
in front of the error patterns stand for the scores adjusted for 
subtraction problems, even though rules were equivocal. The detailed 
description of adjusting criterion is given in Birenbaum & Tatsuoka. The 
point is that the modified dataset improved the approach to 
unidimen8lonality drastically. Moreover, parameter estimation for the 
two parameter logistic model by maximum likelihood (using the computer 
program GETAB written by Robert Bail lie) was processed Successfully with 
the modified dataset while the original dataset failed "^o yield finite 
estimates. 

It is interesting to note that Category 2 contains most erroneous 
patterns with high consistency. On the other hand, Categories 1 and 3 
consist of patterns in which the right rule appears at least once and 
the other elements are zeros. The zeros mean that either the response 
patterns are inconsistent or that rules are so complicated that our 
error-diagnostic system could not determine them specifically. But from 
the error patterns, and a close examination of the generated error 
vectors in our error-diagnostic system, the zeros in Categories 1 and 2 
are mostly due to random errors. Therefore we can safely conclude that 
the structure of the unidimensional subset consists of the responses 
that result from using the right rule, plus random errors. 

Replication of the Result Obtained from the November Data 

The November data were obtained from 8th graders in 1979. A year 
later, a new group of 161 8th graders took a 64-item signed -number 
addition and subtraction cest that w<»8 parallel to the November test. 
The test wad administered right after the teachers (who also taught the 
students in the November data) completed their instruction. The 
teachers used exactly the same teaching method, materials and quizzes as 
those of the last year. The only difference in the new data, the March 
data, is that the numbers in the 16 tasks are slightly changed as shown 
In Appendix V. Table 6 is a summary of the classification of error 
patterns by ICI and total scores. 



Insert Table 6 about here 

The result again indicates that ICI is a useful index for 
extracting a unidimensional dataset by classifying the erroneous 
patterns into Category 2. Also the data structure of the extracted 
unidimensional subset in Table 6 confirms that the responses consist of 
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Table 5 



Error Patterns Causing a Mess in Dimensionality 
of Nine Subtraction Tasks (36 items) 



Category 1 
Error Pattern Frequency 



Category 2 
Error Pattern Frequency 



(1)(1)(1)(D 
0 (D(l)d) 
(1) 0 (1)(1) 
(1)0) 0 (1) 

CD (i)<i) o 

0 0 (1)(1) 
0 (1)(1) o 

(1) 0 (1) 0 
0 (1) 0 (1) 

(1) 0 0 0 
0 (1) 0 0 
0 0 (1) 0 

(1) 0 0 (1) 

(40) (1) (DO) 
0 0 0 0 



15 
5 

3 
2 
2 
2 



Tor.al Wo. of Stu. 41 



#Scores adjusted 

*Performance on addition 
problems was not consistent 

?Scores adjusted for sub- 
traction problems, even 
tnough rule were equivocal 



(5)(5)(5)(5) 
0 (5) (5) (5) 

# 0 0 (5) (5) 

/; (16) (16) (40) (16) 
(5) (40) (5) (5) 

# (16)(16)(5)(16) 

# (16)(16)(16) 0 

# (16) (40) (16) (16) 

# (40) (40) (40) (40) 
(40) 0 (5) (5) 

// 0 (40) (16) (5) 

# (5) (40) (40) (40) 
// (40) (40) (40) (16) 
// 0 (40)(16)(16) 

# (40) (40) (23) (23) 

# (19) (19) (19) (19) 

# 0 (19) (19) (19) 

# 0 0 (40) (40) 
0 (40) 0 0 

# (6) (6) (6) (6) 

I (15) (15) (15) (15) 

# (4) (4) (4) (4) 
// (7) (7) (7) (7) 

// (43) (43) (43) (43) 
f 0 0 0 (5) 



Total lio. of Stu. 



32 



Category 3 
Error Pattern Frequency 



CI) (1) (1XD 
0 (1X1X1) 
(i) o (i)(i) 

(1XD 0 (1) 

(1) (1) (i) o 
0 0 (1)(1) 
0 (1) 0 (1) 

(1X1) o 0 
0 (1X1) 0 

(1) 0 (1) 0 
0 (1) 0 0 
0 0 0 (1) 
0 0 (1) 0 

(40) 0 0 0 
0 0 0 0 



5 
5 
4 

2 



Total No. of Stu. 



Category 4 
Error Pattern Frequency 



t 

It 



0 
0 
0 
0 



0 0 0 

0 0 0 

0 0 (1) 

0 (1) 0 



0 (1) 0 0 

0 0 0 (5) 

0 0 (23) (23) 

0 0 (40) (40) 

0 0 0 (19) 

o o (ii)(i) 

0 (1) 0 (1) 

o (lxixn 

0 (1X1X1) 



Total Wo. of Stu. 



15 
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(1) means that the right rule is used for answering to subtraction problems. 
a Since ICI is calculated over 16 tasks, if performance on addition problems 
is not consistent, then the error pattern falls into Category 3 even though 
the error pattern for subtraction is identical. ^ ^ 
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Table 6 



Replication of the result obtained from the November Data 
N * 161, Classification of Bugs 



Category 1 
Error Pattern Frequency 



Category 3 



Error Pattern 


Frequency 


UXDUXD 


2 


(UU) 0 (1) 


1 


0 (D(l)(l) 


2 


(D(l) 0 0 


2 


0 (1)(1) 0 


7 


0 0 (1)(1) 


2 


(1) 0 (1) 0 


1 


(1) 0 0 (1) 


1 


(1) 0 0 0 


3 


0 (1) 0 0 


1 


0 0 (1) 0 


4 


0 0 0 (1) 


5 


0 0 0 0 


6 


Total No. of Stu. 47 



(1)(1)(1)(1) 

0 CD (1) CD 
(1) 0 (l)(l) 
(D(l) 0 (1) 
CD CD CD 0 
CDCD 0 0 
CD 0 CD 0 
CD 0 0 CD 
0 CDCD 0 
0 0 CDCD 
0 (1) 0 CD 
CD 0 0 o 

0 o 0 CD 
0 0 0 0 
CDClDCDCH) 



32 
7 
5 
6 
4 

1 
x 

1 
1 
3 
2 
1 
1 
1 

1 



Total Ho. of ftr.. 68 



Category 2 


Category 4 


Error Pattern Frequency 


Error Pattern Frequency 


(i)C5)C5) 0 2 


0 (1) 0 (1) 1 


0 C5)C5)C5) 1 


0 0 0 (1) 2 


0 0 C5)'5) 1 


0 0 0 (40) 1 


0 0 C40) 0 1 


0 0 (40) 0 2 


(40) 0 CA0) 0 1 


0 (5) 0 (5) 1 


0 (40) (A.j) 0 2 


0 0 0 0 10 


0 0 0 (40) 2 
(40) (16) (5) (5) 1 




Total No. of Stu. 17 


(40) (16) (40) (5) 1 




0 (5)(16)(16) 1 




0 (16)(16) 0 1 




0 0 0 0 6 




(19)(19) (19) (19) 1 




0 u (19) (19) 1 




0 (19) (19) 0 2 




0 (3) (3) (3) 1 




(11)(11)(U)(11) 1 




0 (43) (43) (43) 1 




(4)(4)C4)(4) 1 




C4) 0 0 (4) 1 





Total No. of Stu. 29 
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answers gotten by using the right rule and of random errors, which is 
the identical result obtained from the November data* 



Insert Table 7 about here 

Table 7 shows the eigenvalues of the March data before and after 
ICI operation was applied. The unidimensionality of the subset improved 
but not as much as that of the November data. Since the proportion of 
the number of subjects in Category 2 is 1/4 in the November data while 
it is 1/8 in the March data, the difference in the increments of the 
variance accounted for by the first eigenvalues in the two datasets may 
be explained. The March data, both the original and those modified by 
ICI yielded convergence when GETAB was used to estimate the parameters 
of the IRT model. 

Conclusion 

One of the important areas in the theory and practice of 
educational measurement is that of criterion-referenced testing. The 
measurement theories that have been utilized in measuring and evaluating 
the outcomes of treatments (or instruction) typically depend on binary 
scores obtained from test items. It was customarily assumed that Che 
underlying structure of the dataset from a criterion-referenced test 
consists of one major common factor because the items are usually 
selected from a single content domain. However, several studies have 
shown that the assumption of unidimensionality should be clo3ely 
examined even for criterion-referenced tests. This study demonstrated 
that the dimensionality of a dataset will be affected by the examinee's 
underlying cognitive processes as well as by the nature of the content 
domain. The fact is that, after the response patterns yielded by 
erroneous rules of operation are deleted from the original dataset, the 
remaining subset of data becomes more nearly unidimensional, and this 
subset of course consists of responses yielded by the right rule and 
non-systematic errors. This observation points to an answer to the 
question of when and why the dataset of an achievement test satisfies 
the condition of unidimensionality. 

Tables 5 and 6 show that ICI is an effective index for datecting 
erroneous rules of oeration. However, it requires repeated measures. 
Since most tests don't have several parallel items to measure the 
performance of a single task, applicability of ICI to these tests will 
be limited. The drawback of this limitation mus be removed, and a 
solution to this problem is being developed. 
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Table 7 

Eigenvalues and their variances of the replicated datasets 



The Original Dataset 



Subset in which low score-high 





M - 161 




ICI subjects are 


deleted 


Factor 


Eigenvalues 


Variance 


Eigenvalue 


Varianc< 


1 


9.2928 


58.0802 


9.4820 


59.2627 


2 


1.7834 


ll.li62 


1.3967 


8.7295 


3 


1.0883 


6.8018 


0.9202 


5.7511 


4 


0.8437 


5.2729 


0.8070 


5.0439 


5 


0.6985 


4.3659 


0.5943 


3.'/ 141 


6 


0.4301 


2.6879 


0.5155 


3.2217 


7 


0.3728 


2.3303 


0.4466 


2.7911 


8 


0.3331 


2.0819 


0.3959 


2.4742 


9 


0.2428 


1.5178 


0.2905 


1.8155 


10 


0.1857 


1.1606 


0.2807 


1.7543 


11 


0.1747 


1.0918 


0.2344 


1.4649 


12 


0.1458 


0.9115 


0.180b 


1.1286 


13 


0.1312 


0.8197 


0.1535 


0.9906 


14 


0.1172 


0.7323 


0.1171 


0.7318 


15 


0.0868 


0.5423 


0.0999 


0.6243 


16 


0.0731 


0.4569 


0.0803 


0.5018 
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Appendices 

I. The signed-number test used in the January experiment 

II. A list of the bugs found in the signed-number addition and 
subtraction problems 

III. Observed complete erroneous rules of operations, their descriptions 
and codes given in Appendix II 

IV. Forty-five rules described in Appendix Illand their component 
scores based on the test in Appendix V 

V. Tue signed-number test used in the March experiment 
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Appendix I 
The Signed-Number Test used in the 
January Experiment 



Test Items 



I 




II 




III 




IV 




1. 


-6-(-3)-2 


17. 


-l-(-10)-9 


33. 


-3-(-5)-2 


49. 


-2-(-ll)-9 


2. 


-7-9—16 


18. 


-2-11 — 13 


34. 


-4-6—10 


50. 


-5-14—19 


3. 


12+-3-9 


19. 


7+-5=2 


35. 


15+-6-9 


51. 


4+-2-2 


4. 


l-(-10)-ll 


20. 


3-(-12)-15 


36. 


5-(-7)-12 


52. 


6-(-8)-14 


5. 


-3+1 2-9 


21. 


-1 M0-9 


37. 


-4+1 3-9 


53. 


-2+11-9 


7. 


8-6-2 


23. 


7-5-2 


39. 


4-2=2 


55. 


9-7-2 


8. 


-16-(-7)— 9 


24. 


-12-(-10)=- 2 


40. 


-ll-(-2)— 9 


56. 


-7-(-5)-2 


10. 


-14+-5— 19 


26. 


-10+- 1—11 


42. 


-7+-5— 12 


53. 


-10+-8— 18 


11. 


3+-5— 2 


27. 


2+-11— 9 


43. 


6+-8— 2 


59. 


1+-10— 9 


12. 


13-(-4)=l7 


23. 


0-(-9)=9 


44. 


6-(-4)-10 


60. 


0-(-2)=2 


13. 


-3-H2— 15 


29. 


-2-+11— 13 


45. 


-7-+9— 16 


61. 


-4-4-6— 10 


15. 


-6+4—2 


31. 


-5+3—2 


47. 


-4+2—2 


63. 


-8+6—2 


16. 


2-11—9 


32. 


5-14—9 


48. 


7-16—9 


64. 


4-13—9 
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Appendix II 

A List of "Bugs" Found in the Sign-numbered Addition and 
Subtraction Problems (Verbal Rules) 





Code 


Incomplete Rule 


(in terms of sign operation) 


1 

1 • 


9 1 
L 1 


Taking 
(i.e., 


the sign 
take the 


of the larger absolute number to answers 
correct sign In addition problems,) 


2. 


22 


Taking 


the sign 


of the 


smaller absolute number to answers 


3. 


23 


Taking 


always a 


+ sign 


to answers 


4. 


24 


Taking 


always a 


- sign 


to answers 


5. 


25 


Taking 


the sign 


of the 


first number 


6. 


26 


Taking 


the sign 


of the 


second number 


7. 


li 


Taking 


the sign 


of the 


product of the two numbers 


8. 


28 


Taking 
answers 


the sign 


of the 


larger number (as an integer) to 


9. 


29 


Taking the $ign 
answers* 


of the 


smaller number (as an integer) to 



10. 210 Treat the operation sign as the sign of the second 

number, if the sign of the second number is explicit. 
Then take the sign of the larger absolute values to 
answers • 

11. 211 Besides the bug described in 210, if the sign of the 

larger absolute value Is the first number *ith an 
implicit sign, then take the explicit sign or the sign 
obtained by the 210 bug to answers 

12. 212 The s^e as bug 210 except ake the sign of the larger integer, 

13. 213 Basically follow the right rule, but when the sign of the 

larger absolute value is implicit and also is the second 
number, then take the sign of the other number to answers* 

(cont. ) 
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Appendix II (cont.) 



Code Incomplete Rule (in terms of sign operation) 



14. 214 Basically follow the bug of code 16 9 but when the larger 

integer is the second number and its sign is implicit, 
then take the sign of the other number* 

15. 215 For the addition problems, if the sign of the second number 

is implicit, then apply the regular arithmetic operation* 
For example, -L+S has a wrong sign* For the subtraction 
problems, the implicit sign of the second number remains 
unchanged when they converted to addition problems, and 
regular arithmetic operation is applied but -L+S type has 
a wrong sign. 

16* 216 Change the explicit sign of the second number, then take 

the sign of the larger absolute value. 

17. 217 Change the explicit sign of the second number, then take 

the sign of the larger integer 

18* 218 Change the sign of the second number and take the sign of 

# the larger absolute value* Treat all problems as 
^subtraction problems. 

19, 219 Change the sign of the second number and take the sign of 

the larger integer value. Treat all problems as subtraction 
problems. 




28 
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Appendix II (cont.) 



Code Incomplete Rules (In terms of absolute value operation) 

1. 11 If the signs of two numbers are same, th ,i add the absolute 

values of the two numbers. If the signs, of two numbers 
are different, then subtract the smaller absolute value 
from the larger absolute value 

2. 12 In spite of different skill types, the absolute values of 

the two numbers are always added. 

3. 13 No matter what the skil~ types are, the smaller absolute 

value is subtracted from the larger absolute value. 

4. 14 Opposite of the right operation, i.e., if the signs 

of two number are same, then subtract, |l| - |s|. 

If the signs of two numbers are different then add the 

absolute values of the two numbers, |l| + |s|. 

5. 15 If the sign of the first number is positive, +, then 

|L| + |S|. 

If the sign of the first number is negative, then 

• Z ILJ - JS |. , 

6. 16 If the sign of the second number is positive, +, then 

\h\ + |s| and if the sign of the second number is 
negative, then |L| - |s|. 

17 If the sign of the first number is positive, then 

|L| - | S I and if i*. is negative, then Jl| + |S|. 

8, 18 If the absolute .alue of the first number is larger, then 

|L| + |SJ and if it is smaller, then |l| - |s| . 

9. 19 If the absolute value of the first number is larger, 

then |L| + |s| . 

10. 110 If the first number is larger as an integer, then 

|L| + |s|. If it is smaller then |l| - |s|. 

11. Ill If the first number is larger as an integer, then |l| - |s| 

and if its not then add the two absolute values. 

12. 112 Changing the sign of the second number, then applying 

the right rule 11. 
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Appendix II (coot.) 
Subtraction Problems 

(Problems in the conversion of subtraction to addition) 

Code 



1. 



2. 



3. 



31 



32 



33 



Convert the operation sign 
of the second number. 

Convert the operation sign 
sign of the second number 

Convert the operation sign 
the first number 



- to + and change the sign 



- to + and don 't change the 



- to + and change the sign of 



5. 



6. 



34 



35 



36 



Convert the operation sign - to + and change both the 
signs of the two numbers 

Convert the operation sign - to + and change ,t|ie sign of 
the second number. At the same time, if the sigh of the 
first number is negative, then it will be changed to + 

Convert the operation sign - to + but don't change the 
sign of the second number. At the same time, if the sign 
of the first number and second numbers are negative, then 
change the signs to +. Thus, task -L - (-S) is converted 
to +L + (+S). 



Oil 



Appendix II (cont.) 



Skill Types 




Conv. 31 


Original 31 




Conv. * 




Conv. 33 


Conv. 34 


Conv. 35 


124-3 


~> 


124-3 


L4-S 




















-3+12 


5 


-3412 


-S4L 




















-144-5 


10 


-144-5 


-L4-S 




















34-5 


11 


34-5 


S4-L 




















-54-7 


14 


-54-7 


-S4-L 




















-6+4 


15 


-644 


-L4S 




















-6- (-8) 


1 


-64(48) 


-S-(-L) ■*■ -S4(4L) 


5 


-S4(-L) 


14 


4S4(-L) 


11 


S4(4L) 


0 


4S4(4L) 


0 


-7-9 


2 


-74-9 


-S-L + -S4-L 


14 


-S4L 


5 


4S4L 


0 


S4-L 


11 


S4-L 


11 


l-(-10) 


4 


14(410) 


S-(-L) > S4(4L) 


0 


S4(-L) 


11 


-S4(-L) 


14 


-S4(4L) 


5 


S4(4L) 


0 


8-'j 


7 


84-6 


L-S -*• L4-S 


3 


L4S 


0 


-L4S 


15 


-L4-S 


10 


L4-S 


3 


-16-(-7) 


8 


-164(47) 


-S-(-S) ■*■ -L4(4S) 


15 


- 4(-S) 


10 


L4(-S) 


3 


4L4(4S) 


0 


L4(4S) 


0 


-12-3 


9 


-124-3 


-L-S ■*• -L4-S 


10 


-L4S 


15 


L4S 


0 


4L4-S 


3 


L4-S 


0 


9-(-7) 


12 


94(47) 


L-(-S) L4(4S) 


0 


L4(-S) 


3 


-L4(-S) 


10 


-L4(4S) 


15 


L4(4S) 


0 


-3-412 


13 


-34-12 


-S-4L - -S4-L 


14 


-S44L 


5 


S44L 


11 


4S4-L 


11 


S4L 


11 


2-11 


16 


24-11 


S-L ■*■ S4-L 


11 


S4L 


0 


-S44L 


14 


-S4-L 


14 


C4-L 


11 


-13-44 




-134-4 


-L-4S ■*■ -L4-S 


10 


-L44S 


10 


L44S 


3 


4L4-S 


3 


L4-S 


3 


13-415 




134-15 


S-4L ■*■ S4-L 


11 


S44L 


0 


-S44L 


5 


-S4-L 


14 


S4-L 


11 



0: L4S - S4L 



31 
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Appendix II (cont.) 

Subtraction Problems 
Having Hidden Signs and Parentheses 
(Interpretation of the problems) 



When the sign of the second number is implicit, the sign 
of the second number is ignored. Thus, the sign of the 
second number remains the same although the operation 
sign f, - M is changed to H + w . This error appears in all 
4 types of problems, L - S, S - L, -L - S and -S - L. 

In the above mentioned case, L -\ S and S - L are carried 
out by regular arithmetic but -L-VS and -S - L will be 
converted to -L + S and -S + L respectively. 

L - S and S - L are treated as regular arithmetic problems. 
The implicit signs of the skill types -L - S, -S - L 
are not ignored. So they are changed to -L + -S 
-S + -L respectively 

The parentheses of the problems are interpreted as absolute 

value symbols before any conversion of subtraction to 

addition* 

The parentheses are treated as absolute value symbols after 
any conversion occurs. 

Besides L - S and S - L types, students apply regular 
arithmetic operations to (-L) - (-S) and (-S) - (-L) 
types and get wrong signs for S - L and/or (-S) - (-L). 

The operation sign in the task -L - S is recognized as the 
sign of the second number. So when a student converts sub- 
traction problems to addition problems, insert a + sign before 
tne second minus sign, right after the larger number L. So this 
task will yield the task type, -L + -S. 
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Appendix III 
Observed Complete Erroneous Rules of Operations 
Their Descriptions and Codes Given in Appendix H 



1. (11,21)(31)(11,2J) 

The right rule for addition problems, (11,21) — add the absolute 
values of two numbers if the signs of the numbers are same, or subtract 
the smaller absolute value from the larger one if the signs are alike 
and take the sign of the larger number to the answer* Then convert 
subtraction problems into addition ones by changing the operation sign 
-to + and the sign of the second number to a opposite sign. Carry out 
the right addition operation on the newly converted additon problems* 

2. (ll,21)(31)Paf(ll,21) 

The right rules for addition problems, and right conversion (31) is 
carried out but the numbers in the parenthesis are changed to positive 
numbers, then 'he right rule for addition is used for the new addition 
problems • 

T. (ll,21)Pbt(31)TlT72T7 - — " — 

The numbers in the parenthesis are changed to positive before 
subtraction problems are converted to addition problems* 

4. (11,25)(31)(11,25) 

xne student takes the right absolute value in answers for addition 
problems. For subtraction problems, the student converts subtraction 
to addition correctly but applies the same wrong rule. 

5. (11,21)(32)(11,21) 

The right rule, for taking the proper absolute value and the sign 
of the larger absolute value, is used for addition problems* For 
subtraction problems a student changes the operation sign of - to + 
without changing 0 the sign of the second number then applies the right 
rule for addition problems* The code is expressed by (11,21) for 
addition, conversion error (32) and the right rule (11,21) again* Thus, 
this br.g is (11,21)(32)(11,21). 

6. (ll,21)(32)a(ll,21) 

The addition problems are right, but subtraction problems are 
converted in a wrong way — by changing operation signs, to plus, +, 
except for the problems types L - S and S - L* The latter two tasks are 
answered by a regular arithmetic method* The other converted tasks are 
answered by using the right rule* 

7. (ll,21)(32)ha(ll,21) 

Apply the right addition rule and converts most subtraction 

problems by applying (32) ~ changes operation signs to plus but doesn't 

change the signs of the second number — but -L - S type. Operation + 

is inserted right after the larger number L* So the problem becomes 
-L + *S type. 
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8. (ll,21)(32)hoa(ll,21) 

This bug is a combination of bugs (U,21)(32)a(U,21) and 
(ll,21)(32)ho(ll,21). That is, -L - S type was changed to -L f -S, and, 
moreover, L - S and S - L types were answered by a regular arithmetic 
method* 

9. (11,21)(36)(11,21) 

After all subtraction problems are converted to addition problems 
according to the rule (36) — convert the operation sign - to +, without 
changing the sign of the second number* At the same time, If there are 
three minuses such as -L-(-S), then all minuses will be plus* Thus 
+L+(+S) is the converted problem type* 

10. (11,21)(33)(11,21) 

Convert subtraction problems into addition problems but change the 
sign of the first number instead of the second number* The right 
addition rule was applied before and after the conversion* 

11* (ll,21)(31)hs(ll,21) 

Whe n subtraction p rnMemfl ar e converted &AM ti nn f hit] A*n signs 

of the second numbers are ignored, so they are not changed to negative* 
Thus -L - S, -S-L, S-L, L-S resulted in wrong answers* 

12. (U,21)(31)hsa(ll,21) 

L-S and S-L types are answered by a regular arithmetic method 
without being converted, but -L-S and -S-L are converted to - L + S 
and -S + L, resulting in wrong answers* 

13. (ll,21)(32)Pbf(ll,21) 

Before subtraction problems are converted, the parenthesis in the 
problems are considered as an absolute value notation and the numbers in 
the parenthesis are changed to positive numbers before the conversion 
(32) is taken • 

14. (ll,21)(32)Paf(ll,21) 

The numbers in the parenthesis are changed to be positive numbers 
after subtraction problems are converted to addition problems according 
to the wrong rule (32). 

15. (ll,21)(32)hoPaf(ll,21) 

-L-S and -S-L types resulted in right conversion even though 
the wrong conversion rule (32) was applied. Then the problems with 
parenthesis are changed to be positive* 

16. (11,20(13,21) {or (11 ,21)(32)(13,21)} 

A student used the right rule for addition problems but he/she 
subtracted two numbers, |L| - |S|, and took the sign of the larger 
absolute value. 
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17. (ll f 21)Paf(13,21) 

After carrying out the rule 13, subtracting two numbers, the 
numbers in the parenthesis are changed to positive numbers then the rule 
(21) is used for taking the sign to answers. 

18. (ll,21)ho(13,21) 

For -L - S type, having a hidden sign for the second number, a 
student treated operation sign - as the sign of the second number. 

19. (12,21)(31)(12,21) 

Subtraction problems are converted to addition problems by the 
right conversion rule but an erroneous rule for addition problem, 
(12,21) are applied consistently both before and after the conversion. 

20. (13,21)(31)(13,21) 

The operation of converting subtractin to addition is carried out 
correctly but a wrong rule (13,21)-- always subtracting two numbers, |L| 
- |S| and taking the sign of the larger absolute value — was applied 
throughout the problems. 



21. (ll,?l)(12,24) 

The conversion of subtraction was not carried out. For addition, 
the right rule was used, but for subtraction problems, two numbers are 
added and a minus sign was taken to the answers. 

22. (11,21)(13,24) 

The conversion of subtraction was not carried out. The rule 
(11,21) for addition, the rule (13,24) for subtraction problems. 

23. (13,21)(13,21) 

The conversion of subtraction was omitted and two numbers are 
always subtracted and the sign of the larger absolute value was always 
taken to answers. 

24. (13,21)(13,24) 

The conversion was omitted and erroneous rules (13,21) for addition 
problems, (13,24) for subtraction problems are used. 

25. (13,21)(32)a(ll,21) 

For addition problems, the wrong rule (13,21) is used. The 
conversion is again wrong (32). L - S and S - L are answered by a 
regular arithmetic method. The rest of the newly converted addition 
problems are answered by the right rule 

26. (13,24)(13,24) 

The same rule, |L| - |S| and the sign, to answers. 

27. (12,24)(13,24) 

The conversion operation is ignored. Add two numbers for addition, 
subtract two numbers |L| - I S | for subtraction problems. The sign, 
is always taken to answers. 

ERIC 35 
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28. (12,23X13,24) 

No conversion Is made. The two numbers are added for addition 
problems and subtracted for subtraction problems. The sign, +, is 
always taken to answers for addition while the sign, is always taken 
for subtraction problems. 

29. (13,23)(13,24) 

Nc conversion is made. The two numbers are always subtracted, |L| 

- I S I but -I- for addition problems and, for problems are taken to 
answers. 

30. (13,25X13,25) 

No conversion is made. The two numbers are always subtracted, 
|L| - | S| and the sign of the first number is taken to answers. 

31. (13,25)(li,24) 

No conversion is made. The two numbers are always subtracted |L| 
* |S|. The sign of the first number is taken to answers for addition 
problems and a minus sign is always taken for subtraction problems. 

32. (13,23X13,23) 

No conversion is made. The two numbers are always subtracted, |L| 

- |S|. The sign, +, is always taken to answers. 

33. (12,25X13,25) 

No conversion is made. The two numbers are added for addition and 
subtracted for subtraction problems. The sign of the first numbber is 
taken to answers for both the addition and subtraction problems. 

34. (15,25X15,25) 

The operation of converting subtraction to addition problems is 
ignored. Instead, this student added two numbers, |L| + |S|, if the 
sign of the first number is positive. He subtracted, |L| - |S| if the 
sign of the first number is negative. He used this rule for both the 
addition and subtraction problems. 

35. (15,25X13,24) 

For addition problems, the same rule as the previous example is 
applied but for subtraction problems, he subtracts two numbers, |L| - |S| 
and takes a minus sign. 

36. (16,23X13,23) 

For addition problems, if the sign of the second number is positive 
then two absolute values are added, |L| + |S| and if it is negative, 
then the two absolute values are subtracted, |L| - |S|. For 
subtraction ^tfclems, the two numbers are always subtracted aid a 
positive sign, +, is always chosen for the sign to answers. 
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37. (16,26)ho(16,26) 

For addition problems, the absolute values of answers are the same 
as the previous example of (16,23). But the sign of the answers la the 
sign of the secord number. For subtraction problems, the same rule as 
addition is used, but -I - S or -S - L types where the sign of the 
second number is implicit, then operation sign is considered as the sign 
of the second number. 

38. (17,21X17.21) 

For addition and subtraction problems, if the sign of the first 
number is +, then |L| - |S| and if it is negative, then |L| + |S|. But 
the sign of the larger absolute value is taken to answers. 

39. (12,21X13,21) 

For addition problems, absolute values of the two numbers are 
always added and the sign of the larger number is taken to answers. For 
subtraction prob' s, two absolute values are always subtracted and the 
sign of the large number is taken to answers. 

40. (12,21)Ai (13,21) 

The right rule for addition problems. For subtraction problems, 
L - S, S - L, -L - (-S) and -S - (-L) types are answered by using a 
regular arithmetic method but the signs of answers are not always correct. 
All other problems are answered by (13,21)-rule. No conversion 
of subtraction to~ad3Ttion problems was made". 

41. (32)(11, 21)(31)(11, 21) 

Subtraction problems are correctly converted by the rule (31), and 
answered correctly by using the right rule of addition problems. But 
addition problems are also converted by changing the sign of the second 
number and answered by the right rule. 

42. (13,21)(31)(12,21) 

In addition problems, the student finds the difference between the 
two numbers and assigns the sign of the larger absolute value to the 
result. In subtraction, converts subtraction to addition by Rule (31) 
which is right, then adds the two absolute values and takes the sign of 
the larger absolute value from the converted addition problems to 
answers • 

43. (11,27X13,24) 

If two numbers have two similar signs, then the two absolute values 
are added and sign of multiplication is taken to the result. If two 
numbers have different signs, then |L| - |S| as .he absolute value and a 
minus sign to the anewer. 

44. (11,21)(34)(11,21) 

In subtraction, the student changes the signs of both numbers as 
well as the operation sign and carries out the right rule for addition 
problems. 

45. (11,21)(36)(13,21) 

In subtraction, the two absolute values are subtracted and the sign 
of the larger number is taken to the result after conversion rule (36) 
is applied. 
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45 Rules Described in Appendix III and Their Components Scores Based on the Test in Appendix V 



item w 


laDN 


3 


4 


5 


6 


7/8 


9 


10 


11 


12 


13 


14 


15 


16 


3 


14+-7-+7 




















































5 ■ 


-4413-49 




















































10 


-164-3—19 




















































11 


2+-B—6 




















































14 


-6+-9— 15 




















































15 


-8+5—3 














* 










































a 


b c 
















































1 


-3-(-7)-+4 


0 


0 0 


1 


1 1 


0 


0 0 


0 


0 0 


0 


0 0 


1 


0 0 


0 


1 0 


1 


1 1 


1 1 1 


0 


0 0 


1 


1 1 


1 


1 1 


0 


1 0 


2 


-2-8—10 


1 


1 1 


0 


0 0 


0 


0 0 


0 


0 0 


1 


1 1 


0 


0 0 


0 


1 0 


0 


0 0 


0 0 0 


0 


0 0 


0 


0 0 


1 


1 1 


0 


0 0 


4 


5-(-12)-417 


0 


0 0 


1 


1 1 


0 


0 0 


0 


0 0 


0 


0 0 


0 


0 0 


0 


1 0 


1 


1 1 


1 1 1 


0 


0 0 


1 


1 1 


1 


1 1 


0 


0 0 


6 


- 1 1-4-0— 19 




1 1 


1 


1 1 


1 


0 0 


1 


0 0 


1 


0 0 


1 


0 0 


0 


1 0 


1 


1 1 


1 1 1 


1 


0 0 


1 


0 0 


1 


0 0 


1 


0 0 


7 


9-4-+5 




1 1 


1 


0 0 


1 


0 0 


1 


1 1 


1 


1 1 


1 


0 0 


0 


1 0 


1 


0 0 


1 1 1 


1 


0 0 


1 


0 0 


1 


1 1 


1 


1 1 


8 


-15- (-9)— 6 




0 0 


1 


1 1 


1 


0 0 


1 


0 0 


1 


0 0 


0 


0 0 


0 


1 0 


1 


1 1 


1 1 1 


0 


1 0 


1 


1 1 


1 


1 1 


1 


1 1 


9 


-13-5—18 




1 1 


1 


0 0 


1 


0 0 


1 


0 0 


1 


1 1 
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a Sign component scores are in the first column 

b Absolute value component scores are in the second column 

c Regular scores (multiplication of the first and second numbers) 

* Response patterns of Aule 40 are obtained by assuming S-L type has a wrong sign* 
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Appendix V 

The Signed-Humber Test 
The Mar en data 



I 




II 




III 




IV 




1. 


-3-(-7)- 


17. 


-2-(-ll)- 


33. 


-2-(-8)- 


49. 


-3-(-12)- 


2. 


, -2-8- 


18. 


-3-15- 


34. 


-3-9- 


50. 


-6-13- 


3. 


14+-7- 


19. 


7+-4- 


35. 


15+-6- 


51. 


9+-3- 


4. 


5-(-12)- 


20. 


6-(-ll)- 


36. 


4-(-9)- 


52. 


4-(-7)- 


5. 


-4+13- 


21. 


-5+8- 


37. 


-3+11- 


53. 


-2+10- 


6. 


-11-+8- 


22. 


-12-+4- 


38. 


-11-+7- 


54. 


-16-+5- 


7. 


9-4- 


23. 


9-6- 


39. 


8-5- 


55. 


6-4- 


8. 


-15-(-9)- 


24. 


-16- (-10)- 


40. 


-13-(-4)- 


56. 


-9-(-5)= 


9. 


-13-5- 


25. 


-5-3- 


41. 


-15-3- 


57. 


-8-4- 


10. 


-16+- 3- 


26. 


-11+-3- 


42. 


_9+_6- 


58. 


-10+-7- 


11. 


2+-8- 


27. 


4+-12- 


43. 


4+-7- 


59. 


1+-11- 


12. 


8-(-6)- 


28. 


6-(-4)- 


44. 


ll-(-l)- 


60. 


12-(-5)- 


13. 


-5-+11- 


29. 


-4-+11- 


45. 


-6-+8- 


61. 


-7-+8- 


14. 


-6+~ 9- 


30. 


-7+-9- 


46. 


-4+-12- 


62. 


-4+-15- 


15. 


-8+5- 


31. 


-7+2- 


47. 


-11+3- 


63. 


-9+5- 


16. 


1-10- 


32. 


6-12- 


48. 


5-13- 


64. 


6-14- 



12 



Navy 



Distribution List 
Navy 



Meryl s. Baker 

NPRDC 

Code P309 

San Diego, CA 92152 

Dr. Jack R« Bor sting 
Provost & Academic Dean 
U.S. Naval Postgraduate School 
Monterey, CA 939*10 

Dr. Robert Breaux 
Code N-711 
NAVTRAEQUIPCEN 
Orlando, FL 32813 

Chief of NavatS^ilcation and Training 

Liason 0ffh?e"* 
Air Force Human Resource Laboratory 
Flying Training Division 
WILLIAMS AFB, AZ 85:4- 

CDR Mike Curran 
Office of Naval Research 
800 #. Quincy St. 
Code 270 

Arlington, VA 22217 

DR. PAT FEDERICO 

NAVY PERSONNEL P«D CENTER 

SAN DIFGO, CA 92152 > 

Mr. Paul Foley 

Navy Personnel R&D Center 

San Dieso f CA 92152 

Dr. John Ford 

Navy Personnel R&D Center 

San Diego f CA 92152 

Dr. Norman J. Kerr 
Chief of Naval Technical Training 
Naval Air Station Memphis (75) 
Millingtor, TN 38051 

Dr. William L. Ma) oj 
Principal Civilian Advisor for 

Education and Training 
Naval Training Command, Code 00A 
Pensaeola, FL 32508 



1 CAPT Richard L. Martin, USN 
Prospective Commanding Officer 
USS Carl Vinson (CVN-70) 
Newport News Shipbuilding and Drydock Co 
Newport News, VA 23607 

1 Dr. James McBride 

Navy Personnel R&D Center 
San Diego, CA 92152 

1 Dr William Montague 

Navy Personnel R&D Center 
San Diego, CA 92152 

'1 Ted M. I. Yellen 

Technical Information Office, Code 201 
NAVY PERSONNEL R&D CENTER 
SAN DIEGO, CA 92152 

1 Library, Code P201L 

Navy Personnel R&D Center 
San Diego, CA 92152 

6 Commanding Officer 

Naval Research Laboratory 
Code 2627 

v;«: Islington, DC 20390 

1 Psychologist , 
0NR Branch Office / 
Bldg 114, Section D 
666 Summer Street 
Boston, MA 02210 

« Psychologist 
0NR West 

1030 East Green St. 
Pasadena, CA 91106 

1 Office of Naval Research 
(442 PT) 

80C N. Quincy Street 
Arlington, VA 22217 

5 Personnel & Training Research Programs 
(442 PT) 
Office of Naval Research 
Arlington, VA 22217 



43 



Navy 



Havy 



1 Psychologist 

OKR Branch Office 
1030 East Green Street 
Pasadena, CA 91101 



1 Dr. Robert Wisher 
Code 309 

Navy Personnel R&D Center 
San Diego, CA 92152 



1 Office of the Chief of Naval Operations 
Research Development & Studies Branch 

' (0P-115) 

Washington, DC 20350 

1 LT Frank C. Petho, MSC, USN (Ph.D) 

Selection and Training Research Division 
Human Performance Sciences Dept. 
Naval Aerospace Medical Research Labor at 
Pensacola, FL 32508 

1 Dr. Bernard Rimland (03B) 
Navy Personnel R&D Center 
San Diego, CA 92152 

1 Dr. Worth Scanland. Director 

Research, Development, Test & Evaluation 
N-5 

Naval Education and Training Command 
NAS, Pensacotta, FL 32508 

1 Dr. Robert G. Smith 

Office of Chief of Naval Operations 
OP-987H 

Washington, DC 20350 

1 Dr. Alfred F. Smode 

Training Analysis & Evaluation Group 

(TAEG) 
Dept. of the Navy 
Orlando, FL 32813 

1 Dr. Richard Sorensen 

Navy Personnel R&D Center 
San Diego, CA 92152 

1 Dr. Ronald Weitzman 
Code 54 WZ 

Department of Administrative Sciences 
U. S. Naval Postgraduate School 
Monterey, CA 939*0 

1 Mr John H. Wolfe 
Code ?310 

U. S. Navy Personnel Research and 

Development Center 
San Diego, CA 92152 



ERJC 44 



DR. MARTIN F. WISKOFF 
NAVY PERSONNEL R& D CENTER 
SAN DIEGO- CA Q21SP 

Army 



1 Technical Director 

U. S. Army Research Institute for the 

Behavioral and Social Sciences 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

1 Dr. Beatrice J. Farr 

U. 3. Army Research Institute 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

1 Dr. Myron Fischl 

U.S. Army Research Institute for the 
Social $nd Behavioral Sciences 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

1 Dr. Dexter Fletcher 

U.S. Army Research Institute 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

1 Dr. Michael Kaplan 

U.S. ARMY RESEARCH INSTITUTE 
5001 EISENHOWER AVENUE 
ALEXANDRIA, VA 22333 

1 Drl Milton S. Katz 

Training Technical Area 
U.S. Army Research Institute 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

1 Dr. Harold F. O f Neil, Jr. 
Attn: PERI-OK 
Army Research Institute 
5001 Eisenhover Avenue 
Alexandria, VA 22333 

1 Mr. Robert Ross 

U.S. Army Research Institute for the 
Social and Behavioral Sciences 
5001 Eisenhower Avenue 
Alexandria, VA 22333 



Air Force 



Marines 



1 Air Force Human Resources Lab 
AFHRL/MPD 

Brooks AFB, TX 78235 

1 U.S. Air Force Office of Scientific 
Research 
Life Sciences Directorate, NL 
Boiling Air Force Base 
Washington, DC 20332 

1 Dr. Earl A. Alluisi 
HQ, AFHRL (AFSC) 
Brooks AFB, TX 78235 

1 Dr. Genevieve Haddad 
Program Manager 
Life Sciences Directorate 
AFOSR 

Boiling AFB, DC 20332 

1 Research and Measurment Division 
Research Branch, AFMPC/MPCYPR 
Randolph AFB, TX 78148 

1 Dr. Malcolm Ree 
AFHRL/MP 

Brooks AFB, T 78235 



1 H. William Greenup 

Education Advisor (E03D 
Education Center, MCDEC 
Quantico, VA 2213M 

1 Major Howard Langdon 

Headquarters, Marine Corps 
OTTI 31 

Arlington Annex 

Columbia Pike at Arlington Ridge Rd . 
Arlington, VA 20380 

1 Director, Office of Manpower Utilization 
HQ, Marine Corps (MPU) 
BCB, Bldg. 2009 
Quantico, VA 22134 

1 Headquarters, U. S. Marine Corps 
Code MPI-20 
Washington, DC 20380 

1 Special Assistant for Marine 
Corps Matters 
Code 100M 

Office of Naval Research 
800 N. Quincy St. 
Arlington, VA 22217 



1 Dr. Marty Rockway 
Technical Director 
AFHRL(OT) 

Williams AFB, AZ 58224 



Army 



1 Dr. Robert Sasmor 

U. S. Army Research Institute for the 

Behavioral and Social Sciences 
5001 Eisenhower Avenue 
A^xandria, VA 22333 

1 Dr. Frederick Steinheiser 
Dept. of Navy 
Chief of Naval Operations 
OP-113 

Washington, DC 20350 

1 Dr. Joseph Ward 

U.S. Army Research Institute 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

ERLC 



1 Major Michael L. Patrow, USMC 
Headquarters, Marine Corps 
(Code MPI-20) 
Washington, DC 20380 

1 DR. A.L. SLAFKOSKY 

SCIENTIFIC ADVISOR (CODE RD-1) 
HQ, U.S. MARINE CORPS 
WASHINGTON, DC 20380 



CoastGuard 



1 Chief, Psychological Reserch Branch 
U. S. Coast Guard (G-P-1/2/TPU2) 
Washington, DC 20593 

1 Dr. Thomas A. Warm 

U.S. Coast Guard Institute 
P.O. Substation IS 
Okla-.ioma City, Oil 73169 



Dr. Halter Bogan 

4615 N. Park Ave, apt. 1611 

Chevy Chase, MD 20015 

Dr. Earl Hunt 
Dept. of Psychology 
University of Washington 
Seattle, HA 98105 

Dr. Jack Hunter 
2122 Coolidge St. 
Lansing, HI 48906 

Dr. Huynh Huynh 
College of Education 
University of South Carolina 
Colunbia, SC 29208 

Professor John A. Keats 
University of Newcastle 
AUSTRALIA 2308 

Mr. Marl in Kroger 
1117 Via Goleta 

Palos Verdes Estates, CA 90274 
Dr. Michael Levine 

Department of Educational Psychology 
210 Education Bldg. 
University of Illinois 
Champaign, IL 61801 

Dr. Charles Lewis 

Faculteit Sociale Wetenschappen 

Rijksuniversiteit Croningen 

Oide Boteringestraat 23 

9712GC Groningen 

Netherlands 

Dr. Robert Linn 
College of Education 
University of Illinois 
Urbana, IL 61 801 

Dr. Frederick M. Lord 
Educational Testing Service 
Princeton, NJ 08540 

Dr. Gary Marco 

Educational Testing Service 

Princeton, NJ 08450 

Dr. David McArthur 
CSE 145 MOOR Hall 
UCLA 

Los Angeles, CA 90024 



ERLC 



4iS 



Harry McMahon 

Miami-Dade Community College 
District Administration 
11011 S.W. 104 Street 
Miami, Florida 33176 

Dr. Scott Maxwell 
Department of Psychology 
diversity of Houston 
Houston, TX 7700U 

Dr. Samuel T. Mayo 
Loyola University of Chicago 
820 North Michigan Avenue 
Chicago, IL 60611 

Dr. Melvin R. Novick 

356 L<ndquist Center for Measurment 

University of Iowa 

Iowa City, IA 52242 

Dr. Jesse Orlansky 
Institute for Defense Analyses 
400 Army Navy Drive 
Arlington, VA 22202 

Wayne M. Patience 
American Council on Education 
GED Testing Service, Suite 20 
One Dupont Cirle, NW 
Washington, DC 20036 

Dr. James A. Paulson 
Portland State University 
P.O. Box 751 
Portland, OR 97207 

MR. LUIGI PETRULL0 
2431 N. EDGEW00D STREET 
ARLINGTON , VA 22207 

DR. DIANE M. RAMSEY-KLEE 
R-K RESEARCH & SYSTEM DESIGN 
3947 RIDGEM0NT DRIVE 
MALIBU, CA 90265 

MINRAT M. L. RAUCH 
P II 4 

BUNDESMINISTERIUM DER VERTEIDICUNG 

PQSTFACH 1328 

D-53 BONN 1, GERMANY 

Dr. David Miller 

Graduate School of Education 

UCLA 

Loa Angeles, CA 90024 

Dr. Anthony J. Nitko 

School of Education 

Division of Educational Studies 

University of Pittsburgh 

5C03 Forbea Quandrangle 

Pittsburgh, PA 15260 



1 Or. Menucha Birenbaua 
School of Education 
Tel Aviv University 
Ramat Aviv P.O. Box 39040 
Tel Aviv 69978 
Israel 

1 Dr. Norman Cliff 
Dept. of Psychology 
Univ. of So. California 
University Park 
Los Angeles, CA 90007 

1 Dr. William E. Coffman 

Director, Iowa Testing frograms 
33^ Lindquist Center 
University of Iowa 
Iowa City, IA 52242 

1 Dr. Allan M. Collins 

Bolt Beranek & Newnan, Inc. 
50 Moulton Street 
Cambridge, Ma 02138 

1 Dr. Meredith P. Crawford 

American Psychological Association 
1200 17th Street, N.W. 
Washington, DC 20036 

1 Dr. .Fritz Drasgow 

Yale School of Organization and Manag 
Yale University 
Box 1A 

New Haven, CT 06520 



1 Dr. Paul A. Games 
403D Carpenter 
University Park, PA 16802 



1 Professor Donald Fitzgerald 
University of New England 
Armidale, New South Wales 2351 
AUSTRALIA 

1 Dr. Edwin A. Fleishman 

Advanced Research Resources Organ 
Suite 900 

4330 East West Highway 
Washington, DC 20014 

1 DR. ROBERT GLASER 
LRDC 

UNIVERSITY OF PITTSBURGH 
3939 0'HARA STREET 
PITTSBURGH, PA 15213 

1 Dr. Bert Green 

Johns Hopkins University 
Department of Psychology 
Charles & 3*lth Street 
Baltimore, M0 21218 

1 Dr. Ron Ham! *eton 
School of Education 
University of Massechusetts 
Amherst, MA 01002 



Dr. Leonard Feldt 

Lindquist Center for Measurment 

University of Iowa 

Iowa City, IA 52242 

Dr. Richard L. Ferguson 

The American College Testing Program 

P.O. Box 168 

Iowa City, IA 522U0 

Dr. Victor Fields 
Dept. of Psychology 
Montgomery College 
Rockville, MD 20850 

Univ. Prof. Dr. Gerhard Fischer 
Liebiggasse 5/3 
A 1010 Vienna 
AUSTRIA 

Dr. Delwyn Harnisch 
ICBD 

University of Illinois 
51 Gerty Drive 
Champaign, II 61801 



ERJC 



17 



Dr. Chester Harris 
School of Education 
University of California 
Santa Barbara, CA 93106 

Dr. Lloyd Humphreys 
Department of Psychology 
University of Illinois 
Champaign, IL 61820 

Library 

Hum RR0/ Western Division 
27857 Berwick Drive 
Carmel, CA 93921 

Dr. Steven Hunkw. 
Department of Education 
University of Alberta 
Edmonton, Alberta 
CANADA 

Dr. Paul Holtzman 
Decision Systems 
MFI 

100 S. Wacker Drive 
Chicago, IL 60606 



Civil Govt 



1 Dr. Susan Chi pro an 

Learning and Development 
National Institute of Educatlo 
1200 19th Street NW 
Washington, DC 20208 

1 Dr. Andrew R. Molnar 
Science Education Dev. 

and Research 
National Science Foundation 
Washington, DC 20550 

1 Dr. Joseph Psotka 

National Institute of Education 
1200 19th St. NW 
Washington, DC 20208 

1 Dr. Vern W. Urry 

Personnel R&D Center 
Office of Personnel Management 
1900 E Street NW 
Washington, DC 20415 

1 Dr. Joseph L. Young, Director 
Memory & Cognitive Processes 
National Science Foundation 
Washington, DC 20550 

Other DoD 



12 Defense Technical Information Center 
Cameron Station, Bldg 5 
Alexandria, VA 22314 
Attn: TC 

1 Dr. William Graham 
Testing Directorate 
MEPC0M/MEPCT-P 
Ft. Sheridan, IL 60037 

1 Military Assistant for Training and 

Personnel Technology 
Office of the Under Secretary of Defense 

for Research 4 Engineering 
Room 3D 129, The Pentagon 
Washir;°*cn, DC 20301 

1 Dr . Wayne Sellman 

Office of the Assistant Secretary 
of Defense (MRA & L) 
2B269 The Pentagon 
Washington, DC 20301 

1 DARPA 

1400 Wilson Blvd. 
O Arlington, VA 22209 A 

ERLC ^ 



Non Govt 

1 Dr. James Algina 

University of Florida 
Gainsville, Fl 32611 

1 Dr. Erling B. Andersen 
Department of Statistics 
Studiestraede 6 
1455 Copenhagen 
DENMARK 

1 Dr. John Annett 

Department of Psychology 
University of Warwick 
Coventry CV4 7AL 
ENGLAND 

1 1 psychological research unit 
Dept. of Defense (Army Office) 
Campbell Park Offices 
Canberra ACT 2600, Australia 

1 Dr. Isaac Be jar 

Educational Testing Service 
Princeton, NJ 08450 

1 Dr . Werner Birke 

DezWPs im Streitkraefteamt 
Postfach 20 50 03 
D-5300 Bonn 2 
WEST GERMANY 

1 Liaison Scientists 

Office of Naval Research, 
Branch Office , London 
Box 39 FP0 New York 09510 

1 Dr. Robert Brennan 

American College Testing programs 

P. 0. Box 168 

Iowa City, IA 52240 

1 DR. C. VICTOR BUNDERS0N 
WICAT INC. 

UNIVERSITY PLAZA, SUITE 10 
1160 SO. STATE ST. 
0REM, UT 84057 

1 Dr. John B. Carroll 
Psychometric Lab 
Univ. of No. Carolina 
Davie Hall 013A 
Chapel Hill, NC 27514 

1 Dr. Deborah Coates 
Catholic University 
620 Michigan Ave. NE 
Washington D.C. 2006A 



ERIC 



Dr. Mark D. Reckase 
Educational Psychology Dept. 
University of tiissouri-Colunbia 
M Hill Hall 
Columbia, MO 65211 

Dr. Lauren Re snick 
LRDC 

University of Pittsburgh 
3939 O f Hara Street 
Pittsburgh, PA 15213 

Dr. Leonard L. Rosenbaun, Chairman 
Department of Psychology 
Montgomery College 
Rockville, MD 20850 

Dr. Ernst Z. Rothkopf 
Bell Laboratories 
600 Mountain Avenue 
Murray Hill, NJ 07974 

Dr. Lawrence Rudner 
M03 Elm Avenue 
Takoma Park, MD 20012 

Dr. J. Ryan 

Department of Education 
University of South Carolina 
Columbia, SC 29208 

PROF. FUMIK0 SAM EJ IMA 
DEPT. OF PSYCHOLOGY 
UNIVERSITY OF TENNESSEE 
KNOXVILLE, TN 37916 

DR. ROBERT J. SEIDEL 
INSTRUCTIONAL TECHNOLOGY GROUP 

HUMRR0 
300 N. WASHINGTON ST. 
ALEXANDRIA, VA 22314 

Dr. Kazuo Shigemasu 

University of Tohoku 

Department of Educational Psychology 

Kawauchi, Sendai 980 

JAPAN 

Paul Ho It man 
Decision Gysteno 
MFI 

100 S. Wacker Drive 
Chicago , IL 60606 

Evelyn Doody 
P.O. Box 1913 
Seaside, CA 03955 

Dr. Drew Malizio 
American Council on Education 
No* 1 Pont Circle, #20 
Washington D.C. 20036 



43 



Dr. Edwin Shirkey 
Department of Psychology 
University of Central Florida 
Orlando, FL 3^816 

Dr. Richard Snow 
School of Education 
Stanford University 
Stanford, CA 9^305 

Dr. Robert Sternberg 
Dept. of Psychology 
Yale University 
Box 11A, Yale Station 
New Haven, CT 06520 

DR. PATRICK SUPPES 

INSTITUTE FOR MATHEMATICAL STUDIES 

THE SOCIAL SCIENCES 
STANFORD UNIVERSITY 
STANFORD, CA 9^305 

Dr. Hariharan Swaminathan 
Laboratory of Psychometric and 

Evaluation Research 
School of Education 
University of Massachusetts 
Amherst, MA 01003 

Dr. Brad Sympson 
Psychometric Research Group 
Educational Testing Service 
Princeton, NJ 08541 

Dr. David 'Thissen 
Department of Psychology 
University of Kansas 
Lawrence, KS 6604M 

Dr. Robert Tsutakawa 
Department of Statistics 
University of Missouri 
Columbia, M0 65201 

Dr . Howard Wainer 
Division of Psychological Studies 
Educational Testing Service 
Princeton, NJ 085M0 

DR. SUSAN E. WHITELY 
PSYCHOLOGY DEPARTMENT 
UNIVERSITY OF KANSAS 
LAWRENCE, KANSAS 66044 

Wolfgang Wildgrube 
Streitkraefteamt 
Box 20 50 03 
D-5300 Bonn 2 

Dr. Steven Wise 

Dept. of Guidance &Educ. Psych. 
- Southern Illinois University 
Carbondale, II 62901 



IN 



